Adjusting a consumer experience based on a 3d captured image stream of a consumer response

ABSTRACT

A response processing system captures a three-dimensional movement of the consumer within a consumer environment, wherein the three-dimensional movement is determined using at least one image capture device aimed at the consumer. The response processing system identifies at least one behavior of the consumer in response to at least one stimulus within the consumer environment from a three-dimensional object properties stream of the captured movement. A consumer interaction system detects whether the at least one behavior of the consumer indicates a type of response to the at least one stimulus requiring adjustment of the consumer environment. Responsive to detecting that the behavior of the consumer indicates a type of response to the at least one stimulus requiring adjustment of the consumer environment, the consumer interaction system generates a control signal to trigger at least one change of the at least one stimulus within the consumer environment.

TECHNICAL FIELD

The present invention relates in general to improved behavioridentification. In particular, the present invention relates toadjusting, within a consumer environment, the consumer experience withinthe environment based on a three-dimensional (3D) captured image streamrepresentative of the consumer response to at least one stimulus withinthe environment.

DESCRIPTION OF THE RELATED ART

Many consumer environments now include multiple different types ofcomputer interfaces specified to request particular types of informationfrom a consumer and specified to provide particular types of products orservices to a consumer.

In one example of a computer interface within a consumer environment,many businesses are starting to implement self-service “kiosks” within astore front, where the self-service kiosk is a computer systemspecifically designed to provide a consumer within a particular productor service within a store environment. For example, kiosks are nowimplemented in store environments to provide for self-service checkout,self-service registry lookup, and self-service banking.

In another example of a computer interface within a consumerenvironment, at check-out counters, many businesses now implement aninterface accessible to the consumer at which the consumer swipes acredit card, enters security information, provides a signature, andresponds to other information requests. Thus, for a consumer to makepurchases, the consumer is typically directed through multiple displayscreens of information, where the consumer must respond to a question oneach screen to complete a purchase.

Different consumers may respond to different products and serviceswithin a consumer environment, and in particular kiosks or othercomputer interfaces, in different ways. For example, some consumers maybecome frustrated or confused when directed to and attempting to use aself-service kiosk within a store. Other consumers, however, may have amore enjoyable shopping experience when those consumers are providedwith a self-service kiosk option.

Additionally, with the expansion of on-line shopping, a consumershopping environment may extend into a consumer's home or anywhere thatthe consumer accesses a store website via a network connection. On-linestore fronts and on-line checkout requirements vary from site to site,often requiring a consumer to learn a new navigation and checkout systemfor each on-line vendor that a consumer visits.

Some companies who provide computer interfaces within a storeenvironment, will program the software running the computer interfacesto prompt a clerk or a help tool when the software detects the consumernot making selections or making selections in error. For example, at aself-service checkout kiosk in a grocery store, if a consumer swipes aproduct over a UPC scanner, but does not place the product in a bag, theself-service checkout kiosk alerts a store clerk to the error.

Responding to a consumer using an interface merely based on theselections made by the consumer at the interface is a limited method ofmonitoring a consumer's experience within an environment. Therefore, inview of the foregoing, there is a need for a method, system, and programproduct for monitoring actual consumer behavior within an environment,detecting which product, service, or other stimulus the consumerbehavior is associated with, and determining what type of response aconsumer has to the stimulus within the environment based on thedetected consumer behaviors, such that in response to a particularconsumer behavior, a clerk may be prompted to intervene, the contentdisplayed on a computer interface may be adjusted, or an additionalcomputer interface may be opened to provide service to a consumer. Inparticular, there is a need for a method, system, and program productfor monitoring consumer behaviors within the environment based on 3Dcaptured image streams, such that consumer behavior is more accuratelypredicted based on a fine granularity, 3D image stream.

SUMMARY OF THE INVENTION

Therefore, the present invention provides improved behavioridentification from a three-dimensional captured image. In particular,the present invention provides for adjusting, within a consumerenvironment, the consumer experience within the environment based on a3D captured image stream representative of the consumer response to atleast one stimulus within the environment.

In one embodiment, a computer-implemented method, system, and programproduct provides for adjusting a consumer experience within a consumerenvironment. A response processing system captures a three-dimensionalmovement of the consumer within the consumer environment, wherein thethree-dimensional movement is determined using at least one imagecapture device aimed at the consumer. The response processing systemidentifies at least one behavior of the consumer in response to at leastone stimulus within the consumer environment from a three-dimensionalobject properties stream of the captured movement. A consumerinteraction system detects whether the at least one behavior of theconsumer indicates a type of response to the at least one stimulusrequiring adjustment of the consumer environment by comparing the typeof response indicated by the at least one behavior with a plurality ofresponse rules and identifying at least one particular response ruleapplicable for the type of response to the at least one stimulus.Responsive to detecting that the behavior of the consumer indicates atype of response to the at least one stimulus requiring adjustment ofthe consumer environment, the consumer interaction system generates acontrol signal according to the particular response rule to trigger atleast one change of the at least one stimulus within the consumerenvironment.

In capturing the three-dimensional movement of the consumer, theresponse processing system captures the three-dimensional movement usinga stereoscopic image device to identify and track a particularthree-dimensional movement. Additionally, in another embodiment, theresponse processing systems captures the three-dimensional movement ofthe consumer using at least one stereoscopic image device and at leastone sensor enabled device to detect content accessible to a consumer atan interface within the consumer environment, wherein the content is thestimulus within the consumer environment.

In identifying at least one behavior of the consumer in response to atleast one stimulus within the consumer environment from athree-dimensional object properties stream, the response processingsystem calculates a percentage probability that the capturedthree-dimensional movement represents a particular behavior defined inthe behavior definitions. Additionally, in identifying at least onebehavior of the consumer in response to at least one stimulus within theconsumer environment from a three-dimensional object properties stream,the response processing system identifies at least one of body movementof the consumer, including fine granularity movement such as facialexpressions, eye movements, muscle contractions, and changes in skinsurface characteristics of the consumer within the three-dimensionalobject properties stream matching at least one of the plurality ofbehavior definitions.

In one embodiment, the consumer environment comprises at least one ofthe consumer within a first selected area including a point of saleterminal, within a second selected area include a display or selectionof items within a store, within a third selected area include a kioskwithin a store, and accessing an online store interface. The stimulusmay include a product selection that a consumer views, picks up, trieson, or otherwise interacts with. Responsive to detecting the consumerwithin the consumer environment, the response processing system capturesthe three-dimensional movement of the consumer within the consumerenvironment.

In generating the control signal, in one embodiment the consumerinteraction system generates the control signal according to theparticular response rule to trigger a program controller to adjust thecontent displayed on an interface within the consumer environment. Inanother embodiment, the consumer interaction system generates thecontrol signal according to the particular response rule to trigger astore attendant system to prompt at least one store attendant within theconsumer environment to provide additional aid to the consumer. In yetanother embodiment, the consumer interaction system generates thecontrol signal according to the particular response rule to triggerrunning an additional computer system within the consumer environment toprovide a service to the consumer.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself however, as well as apreferred mode of use, further objects and advantages thereof, will bestbe understood by reference to the following detailed description of anillustrative embodiment when read in conjunction with the accompanyingdrawings, wherein:

FIG. 1 is a block diagram illustrating a flow of information in aconsumer response processing method, system, and program;

FIG. 2 is an illustrative block diagram depicting an example of anenvironment in which a 3D object detector captures and generates the 3Dobject properties representative of captured behavior movement andstimulus within the consumer environment;

FIG. 3 is a block diagram illustrating one embodiment of a 3D objectdetector system for generating 3D object properties for enabling aresponse interpreter to interpret, from 3D object properties, behaviorsof consumers in response to one or more stimulus within a consumerenvironment;

FIG. 4 is a block diagram depicting one embodiment of a responseinterpreter system;

FIG. 5 is a block diagram illustrating one embodiment of a computingsystem in which the present invention may be implemented;

FIG. 6 is a block diagram depicting one example of a distributed networkenvironment in which a response-enabled method, system, and program maybe implemented;

FIG. 7 is a block diagram illustrating one example of an implementationof a response interpreter system communicating with a response-enabledconsumer interaction system;

FIG. 8 is a block diagram depicting one example of an object detectioninterface and a consumer interaction interface within a storeenvironment;

FIG. 9 is an illustrative diagram illustrating one example of a consumerinteracting with an online consumer environment;

FIG. 10 is a block diagram illustrating a survey controller foraggregating consumer responses to a particular stimulus;

FIG. 11 is a high level logic flowchart illustrating a process andprogram for a response-enabled operating system for adjusting contentbased on consumer behavior responsive to operations controlled by theoperating system;

FIG. 12 is a high level logic flowchart depicting a process and programfor a response processing system to predict response types and stimulusresponded to with a percentage probability;

FIG. 13 is a high level logic flowchart illustrating a process andprogram for response detection by tracking objects within image streamsand other sensed data and generating 3D object properties for thetracked objects representative of behaviors and generating consumerinterface records indicative of any content accessible to the consumerat an interface within the consumer environment

FIG. 14 is a high level logic flowchart depicting a process and programfor response prediction from tracked 3D object properties and consumerinterface records; and

FIG. 15 is a high level logic flowchart depicting a process and programfor applying a predicted behavior record in a response-enabled consumerinteraction system.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

With reference now to FIG. 1, a block diagram illustrates a flow ofinformation in a consumer response processing method, system, andprogram. It will be understood that FIG. 1 provides one embodiment ofinformation flow for capturing image streams within a consumerenvironment and processing those image streams to predict behaviors ofconsumers within the consumer environment, to predict the stimulusresponded to by consumers within the consumer environment, and thus topredict the behavior based response by a consumer to the stimulus,however, other information flows may be implemented to process captureddata and predict behaviors, predict stimulus responded to and thereforepredict a response by a consumer to a stimulus within the consumerenvironment

It is important to note that as used throughout, the term “behavior” mayinclude consumer actions typically considered behavior or body language,such as gesturing and facial expressions, and may also include anydetectable body movements, body posture, changes in skin surfacecharacteristics such as color, temperature, tone, and level ofperspiration, and other types of non-verbal communication. While theembodiment refers throughout to a “consumer”, a consumer may include anyperson interacting with or using a product or service or waiting to usea product or service within a consumer environment. An “environment” or“consumer environment” may include, for example, a selected area inwhich the user accesses a product or service or waits to access aproduct or service. A “consumer environment” may include a storelocation or an online store accessed at a computer system via a network.A “stimulus” may include, but is not limited to, a product, service,store attendant, or other entity within a consumer environment to whicha behavioral response by a consumer may be triggered. Detectedcharacteristics of a stimulus, such as a product, may include, but arenot limited to, a type of product, a brand of product, a size ofproduct, a price of the product, a composition of the product, thenumber of times a consumer interacts with a product, and textualinformation affixed to a product.

In the example, a behavior processing system 100 includes athree-dimensional (3D) object detector 104. 3D object detector 104represents multiple systems for capturing images and other data aboutmoving and stationary objects, streamlining the captured data, trackingparticular objects within the captured movement, streaming theproperties of the particular objects, and combining the streamedproperties into a three-dimensional representation of the 3Dcharacteristics of the captured objects, as illustrated by 3D objectproperties 110. 3D object properties 110 may include, but are notlimited to, positions, color, size, and orientation, representative ofmovement by consumers, objects within an environment, and the backgroundidentifying attributes of an environment, for example.

In the example, 3D object detector 104 captures images within a focusarea, represented as detectable behavior movement 102. Detectablebehavior movement 102 may represent all or a portion of a consumerenvironment. In addition, images within detectable behavior movement 102may include, but is not limited to, images of a consumer, a storeattendant, a product, a service, other entities within a consumerenvironment, and images not representative of a consumer environment.

In particular, 3D object detector 104 detects detectable behaviormovement 102 through multiple types of image and data detectionincluding, but not limited to, capturing video images, detecting bodypart movement, detecting eye movement, detecting skin texture, detectingskin surface characteristics, and capturing thermal images. Forsupporting multiple types of image and data detection, 3D objectdetector 104 may include multiple types of image capture devices,including one or more video cameras arranged for stereoscope video imagecapture, and other types of sensors for capturing depth characteristicsor other types of characteristics of one or more objects. For examples,sensor such as thermal body imaging sensors, skin texture sensors, lasersensing devices, weather detection systems, sound navigation and ranging(SONAR) devices, or synthetic laser or sonar systems, may beimplemented. In particular, a sensor may be implemented using a separatesensor unit or may be implemented through a logic unit that operates ona captured image stream. For example, a logic unit may process thecaptured image stream to detect facial skin textures distinguishablefrom non-skin textures, such as a smooth wall or textured foliate,within a focus area.

3D object detector 104 translates detectable behavior movement 102 intoa stream of 3D properties of detected objects and passes the stream of3D object properties 110 to response interpreter 106. Responseinterpreter 106 maps the streamed 3D object properties 110 into one ormore behaviors for each consumer and estimates, for each predictedbehavior of a consumer, the probability that the actual behavior indetectable behavior movement 102 is correctly predicted by responseinterpreter 106.

In addition, response interpreter 106 detects what stimulus a consumeris responding to with the predicted behavior. In one example, responseinterpreter 106 may detect the stimulus responded to from objectsdetected within 3D object properties 110. In another example, responseinterpreter 106 may detect the stimulus responded to from a consumerinterface record 114 indicating the current content of an interfaceaccessible to a consumer and any inputs by the consumer at theinterface.

Further, in one example, 3D object detector 104 includes a sensorenabled to detect the current content of an interface accessible to aconsumer and generates consumer interface record 114. In anotherexample, response interpreter 106 receives consumer interface record 114from a computer program or other sensor in the form of a record of thecurrent content of an interface accessible to a consumer with a timestamp, and response interpreter 106 matches the time stamp of consumerinterface record 114 with the same time stamp on 3D object properties110.

Further, based on the predicted behavior and the product responded to,response interpreter 106 may also predict, with a percentageprobability, whether the consumer's response to the stimulus is one thatis indicative of a consumer problem. Alternatively, a consumerinteraction controller, as will be further described, may determinewhether a consumer's response to a stimulus requires attention andcontrol outputs to trigger the required attention.

Response interpreter 106 outputs each predicted behavior, percentageprobability that the behavior is correctly predicted, and stimulusresponded to in the behavior as predicted response output 108. Responseinterpreter 106 may pass predicted response output 108 to one or moreconsumer response-enabled applications at one or more systems.

In particular, in processing detectable behavior movement 102 andgenerating predicted response output 108, 3D object detector 104 andresponse interpreter 106 may access a definition database 112 ofpreviously accumulated and stored behavior definitions and stored objectdefinitions to better track and detect, within 3D object properties 110,those monitored objects representative of behaviors, to better recognizeconsumers separate from other objects within detectable behaviormovement 102, to better track and recognize products or services withindetectable behavior movement 102, and to better predict the connectionbetween a consumer behavior and a stimulus responded to by a consumer.

Further, in processing behavior movement 102 and generating predictedresponse output 108, 3D object detector 104 and response interpreter 106attempt to identify objects representative of monitored user behaviorsand to identify objects representative of stimulus and responseinterpreter 106 predicts the type of behavior in view of the overallinteraction in which the behavior is made. Thus, 3D object detector 104and response interpreter 106 attempt to determine not just a behavior bya consumer, but a level of emphasis included in a behavior that wouldeffect the meaning of the behavior, the stimulus responsive to which thebehavior was made that would effect the meaning of the behavior, abackground of a consumer making a behavior that would effect the meaningof the behavior, the environment in which the consumer makes thebehavior that would effect the meaning of the behavior, combinations ofbehaviors made together that effect the meaning of each behavior andother detectable factors that effect the meaning of a behavior. Thus,definition database 112 includes behaviors and object definitionscorresponding to different types of people, environments, and otherfactors that may effect the meaning of a behavior. In addition,definition database 112 includes behavior definitions adjusted accordingto a corresponding facial expression or other corresponding behavior.Further, definition database 112 may be trained to more accuratelyidentify objects representing particular people, animals, places, orthings most common within a particular consumer environment.

In addition, in processing behavior movement 102, multiple separatesystems of image capture devices and other sensors may each captureimage and data about separate or overlapping focus areas from differentangles. The separate systems of image capture devices and other sensorsmay be communicatively connected via a wireless or wired connection andmay share captured images and data with one another, between 3D behaviordetectors or between response interpreters, such that with thecombination of data, response interpreter 106 may interpreter behaviorswith greater accuracy.

Referring now to FIG. 2, an illustrative diagram depicts an example ofan environment in which a 3D object detector captures and generates the3D object properties representative of captured behavior movement andstimulus within a consumer environment. It will be understood thatdetectable movement environment 200 is one example of an environment inwhich 3D object detector 104 detects images and data representative ofdetectable behavior movement 102, as described with reference toresponse processing system 100 in FIG. 1. Other environments may beimplemented in which behavior and other movement indicative of aconsumer response to a consumer environment is detected and processed.

In the example, detectable movement environment 200 includes astereoscopic image device comprising an image capture device 202 and animage capture device 204, each positioned to detect movement of one ormore objects, including consumer movement, within a combined 3D focusarea 220. In the depicted embodiment, image capture device 202 and imagecapture device 204 may each be positioned on one stationary axis orseparate stationary axis, such that the area represented by 3D focusarea 220 remains constantly focused upon. In addition, in the depictedembodiment, image capture device 202 and image capture device 204 andany other sensors may be positioned in parallel, at tangents, or at anyother angles to control the span of and capture images within 3D focusarea 220.

In another embodiment, image capture device 202 and image capture device204 may each be positioned on a position adjustable axis or the actualfocus point of image capture device 202 and image capture device 204 maybe adjustable, such that the area represented by 3D focus area 220 maybe repositioned. In one example, each of image capture device 202 andimage capture device 204 may be coupled with one or more thermal imagingdevices that detect thermal imaging based movement within a broad areaand directs the repositioning of the focus area of each of image capturedevice 202 and image capture device 204 to track the thermal movementwithin the focus area of each camera.

Further, in the present embodiment, image capture device 202 and imagecapture device 204 may be affixed to an apparatus that is carried by orworn by a user such as, but not limited to, a consumer or a storeattendant. For example, image capture device 202 and image capturedevice 204 may be affixed to a pair of glasses or other headwear for auser, such that 3D focus area 220 changes as the user moves. In anotherexample, image capture device 202 and image capture device 204 may beaffixed to a moving machine, such as a robot, such that 3D focus area220 changes as the robot moves.

Although not depicted, in another embodiment, only a single videocamera, such as image capture device 202, may be implemented as astereoscopic image device. The single video camera is placed on a trackor other adjustable axis and a controller adjusts the position of thesingle video camera along the track, wherein the single video camerathen captures a stream of video images within a focus area at differentpositioned points along the track and 3D behavior detector 104 combinesthe stream of images into a 3D object property stream of the propertiesof detectable objects. In one example, the 3D object property stream canbe generated from comparing the changes in luminance and shadowingacross the frames as the camera changes in position. Alternatively, astereoscopic image device may be implemented using a single fixed cameracoupled with a sensor that detects depth. In addition, alternatively, asingle camera enabled to process images and detect depth from a fixedposition may function as a stereoscopic image device. For example, thesingle camera may process images and detect depth from detecting themovement of a light source and comparing changes in luminance andshadowing across the captured image frames. In particular, the singlecamera system may first map a model of a monitored user face, focusingon the eyes, mouth, and nose and then detect changes in luminance andshadowing across image frames to detect depth characteristics of theface. In other examples, a system may process a captured stream of videoimages to extract depth from other characteristics of the stream ofimages.

For purposes of example, 3D focus area 220 includes a first captureplane 206, captured by image capture device 202 and a second captureplane 208, captured by image capture device 204. First capture plane 206detects movement within the plane illustrated by reference numeral 214and second capture plane 208 detects movement within the planeillustrated by reference numeral 216. Thus, for example, image capturedevice 202 detects movement of an object side to side or up and down andimage capture device 204 detects movement of an object forward andbackward within 3D focus area 220. It is important to note that when themovement of an object is tracked at a fine granularity, even smalladjustments in the body movement such as a raised eyebrow, a jaw thrustforward, a smile, or a frown, of a consumer are tracked and can then beinterpreted as indicative of a response by the consumer to a particularstimulus within an environment.

In the example, within 3D focus area 220, a hand 210 represents a movingobject and a box 212 represents a stationary object. In the example,hand 210 is the portion of a person's hand within 3D focus area 220. Amonitored user may make any number of movements, some representative ofbehavior in response to box 212 and others responsive to other stimulus,by moving hand 210.

As a person moves hand 210 within 3D focus area 220, each of imagecapture device 202 and image capture device 204 capture a video streamof the movement of hand 210 within capture plane 206 and capture plane208. From the video streams, 3D object detector 104 detects hand 210 asa moving object within 3D focus area 220 and generates a 3D propertystream, representative of 3D object properties 110, of hand 210 over aperiod of time.

In addition, a user may move hand 210 in relation to box 212 or anotherobject. For example, a user may point or make another type of behaviordirected to box 212. As the user moves hand 210 within 3D focus area220, the video streams captured by image capture device 202 and imagecapture device 204 include the movement of hand 210 and box 212. Fromthe video streams, 3D object detector 104 detects hand 210 as a movingobject and box 212 as a stationary object within 3D focus area 220 andgenerates 3D object property streams indicating the 3D properties of box212 and the 3D properties of hand 210 in relation to box 212 over aperiod of time.

It is important to note that by capturing different planes of movementwithin 3D focus area 220 using multiple cameras, more points of movementare captured than would occur with a typical stationary single camera.By capturing more points of movement from more than one angle, 3D objectdetector 104 can more accurately detect and define a 3D representationof stationary objects and moving objects, including consumer behaviorsand stimulus, within 3D focus area 220. In addition, the more accuratelythat 3D object detector 104 defines a 3D representation of a movingobject, the more accurately behavior interpreter 106 can predict abehavior from the 3D model. For example, a behavior may consist of auser making a motion directly towards or away from one of video camera202 and video camera 204 which would not be able to be captured in a twodimensional frame; 3D behavior detector 104 detects and defines a 3Drepresentation of the behavior as a moving object and behaviorinterpreter 106 predicts the behavior made by the movement towards oraway from a video camera from the 3D model of the movement.

For example, many consumers waiting in a line to check-out may initiallyappear patient, with little facial expression. Most consumers, however,will start to show signs of impatience through small changes in facialexpression. For example, a consumer starting to become impatient maystart to purse the consumer's lips more tightly. By capturing, mapping,and tracking a 3D image of the facial expressions and other behaviors ofconsumers waiting in line at a fine granularity, the adjustment to theamount of pursing of a consumer's lips is detected, such that responseinterpreter 106 may more accurately map to and predict a detectedbehavior is pursed lips which is an initial indicator of impatience andmore accurately predict when the pursed lips reach a level of tightnessrequiring additional assistance to the consumer individually or to allconsumers by opening an additional check-out location.

In addition, it is important to note that while FIG. 2 illustrates agesturing hand 210 and a stationary box 212, in alternate embodiments,3D focus area 220 may include multiple separate consumers storeattendants, or stimulus such that image capture device 202 and imagecapture device 204 capture images of the behavior of multiple people,and 3D object detector 104 detects each behavior by each person as aseparate object. In particular, 3D object detector 104 may detect, fromthe captured video images from image capture device 202 and imagecapture device 204, behaviors with more motion, such as behaviors madewith hands, and behaviors made with less motion, such as facialexpressions, to accurately generate 3D object properties of a person'snon-verbal communication and interaction with others.

In the example, in addition to capturing images within focus area 220,within detectable movement environment 200, other sensors may detectinformation relevant to a consumer environment. For example, sensor 240may detect information within a sensor area 242. Sensor area 242 mayoverlap, be incorporated within, incorporate, or be separate from focusarea 220. 3D object detector 104 combines the sensed information withcaptured images to more accurately generate 3D object properties 110 andto provide additional information about an environment to a warningsystem.

In one example, sensor 240 may detect the content displayed to aconsumer at a particular interface, detect the particular program orparticular program mode running and displayed to a consumer at aparticular interface, or detect other types of information about thedisplay and the underlying applications, operating system, networktransmissions, hardware, or other components of a system that enables aninterface accessible to a consumer. By accessing information about thedisplayed content or the underlying components of an interfaceaccessible to a consumer, sensor 240 detects information that enables 3Dobject detector to generate consumer interface record 114. In anotherembodiment, additional or alternate systems may detect the informationfor enabling 3D object detector to generate consumer interface record114.

In one example, sensor 240 may perform facial recognition or otheridentification recognition from the captured image streams. In oneexample, sensor 240 may access a database of facial mappings formonitored users and supervising users and identify a particular userfacial mapping matching a particular facial mapping from the database.In another example, sensor 240 may analyze the captured image stream fortext that may identify a monitored user. For example, sensor 240 maydetect a badge number the captured image stream of a badge worn by amonitored user or supervising user. By sensor 240 detecting a useridentity, object detector 104 may more accurately generate 3D objectproperties and response interpreter 106 may more accurately predicttypes of user behavior from definitions specified for the identifieduser in definition database 112.

Additionally, in another example, sensor 240 may detect informationbroadcast from RFID chips placed on items within sensor area 242, wherethe RFID of an object broadcasts the object type, the object location,and any information associated with the object. By combining sensedinformation about the location of a particular object with capturedimages from which the image is identified, object detector 104 may moreaccurately generate 3D object properties 110 and response interpreter106 may more accurately distinguish stimulus, such as products, from thebehaviors within 3D object properties 110.

In yet another example, sensor 240 may track the relative location of atracked object within sensor area 242. Although not depicted, sensorarea 242 may track a moving object, including a consumer, a product or aservice, from a first focus area 220 to a second focus area withinsensor area 242. By tracking movement across multiple focus areas,sensor 240 provides additional tracking information of a location of aconsumer, a product, or a service so that data gathered in differentfocus areas can be shared when generating 3D object properties 325.Additionally, sensor area 242 may represent a particular security area,where if motion is detected within sensor area 242, sensor 240 triggersadjustment of focus area 220 to capture the motion-causing entity.Moreover, sensor 240 may detect a changing GPS location of sensor area242 such that maps or other information associated with a particular GPSlocation can be accessed and used in generating 3D object properties 110and consumer interface record 114.

In an additional example, sensor 240 may detect additional informationabout the weather within sensor area 242. In one example, sensor area242 may include an area outside a store front and an area within thestore location. By detecting the weather, including the temperature,outside a store front and within the store location, sensor 242 detectsinformation that may aid response interpreter 106 in predicting thestimulus of consumer behavior. For example, if a consumer enters a storeshivering and sensor 242 provides data indicating a cold temperatureoutside, response interpreter 106 may interpret behaviors by theconsumer in view of the consumer entering the store on a cold day. Inanother example, where a consumer accesses an online store environment,sensor 242 may be a tool running at the computer system accessing theonline store, where the tool detects the current weather for the city orarea in which the computer system is located.

In a further example, sensor 240 may detect additional information aboutthe depth, surface area, color temperature or other characteristic of anobject to more accurately predict whether the object is representativeof a particular behavior and what type of response a consumer has to astimulus within the consumer environment based on the detected behavior.Sensor 240 may gather independent data about an object or may processthe images captured by image capture device 202 and image capture device204. In particular, by detecting additional information about the depth,surface area, or other characteristic of an object, data collected bysensor 240 is combined with images captured by image capture device 202and image capture device 204 to generate additional detail andgranularity in a 3D object.

With reference now to FIG. 3, a block diagram illustrates one embodimentof a 3D object detector system for generating 3D object properties forenabling a response interpreter to interpret, from 3D object properties,behaviors of consumers in response to one or more stimulus within aconsumer environment. It is important to note that the multiplecomponents depicted within 3D object detector 104 may be incorporatedwithin a single system or distributed via a network, other communicationmedium, or other transport medium across multiple systems. In addition,it is important to note that additional or alternate components fromthose illustrated may be implemented in 3D object detector 104 forcapturing images and data and generating a stream of 3D objectproperties 110 and consumer interface record 114.

Initially, multiple image capture devices, such as image capture device202, image capture device 204 and sensor 240, represent a stereoscopicimage device for acquiring the data representative of detectablebehavior movement 102 within a 3D focus area and sensor area, such as 3Dfocus area 220 and sensor area 242. As previously described, imagecapture device 202 and image capture device 204 may represent videocameras for capturing video images. In addition, image capture device202 and image capture device 204 may represent a camera or other stillimage capture device. In addition, image capture device 202 and imagecapture device 204 may represent other types of devices capable ofcapturing data representative of detectable behavior movement 102. Imagecapture device 202 and image capture device 204 may be implemented usingthe same type of device or different types devices. In addition, thescope, size, and location of the capture area and plane captured by eachof image capture device 202 and image capture device 204 may vary.

Sensor 240 may represent one or more different types of sensors aspreviously described with reference to FIG. 2. In particular, sensor 240detects information about objects in a particular sensor area thatenhances the ability of 3D object detector 104 to create 3D objectproperties 110 and consumer interface record 114

Each of image capture device 202, image capture device 204, and sensor240 transmit captured images and data to one or more computing systemsenabled to initially receive and buffer the captured images and data. Inthe example, image capture device 202 transmits captured images to imagecapture server 308, image capture device 204 transmits captured imagesto image capture server 310, and sensor 240 transmits captured data tosensor server 312. Image capture server 308, image capture server 310,and sensor server 312 may be implemented within one or more serversystems.

Each of image capture server 308, image capture server 310, and sensorserver 312 streams the buffered images and data from image capturedevice 202, image capture device 204, and sensor device 240 to one ormore processors. In the example, image capture server 308 streams imagesto a video processor 316, image capture server 310 streams images to avideo processor 318, and sensor server 312 streams the sensed data tosensor processor 319. It is important to note that video processor 316,video processor 318, and sensor processor 319 may be implemented withinone or more processors in one or more computer systems.

In one example, image capture server 308 and image capture server 310each stream images to video processor 316 and video processor 318,respectively, where the images are streamed in frames. Each frame mayinclude, but is not limited to, a camera identifier (ID) of the imagecapture device, a frame number, a time stamp and a pixel count.

Video processor 316, video processor 318, and sensor processor 319 areprogrammed to detect and track objects within image frames. Inparticular, because video processor 316, video processor 318, and sensorprocessor 319 receive streams of complex data and process the data toidentify three-dimensional objects, including objects representingconsumers and movement by consumers, and characteristics of thethree-dimensional objects, video processor 316, video processor 318, andsensor processor 319 may implement the Cell Broadband Engine (Cell BE)architecture (Cell Broadband Engine is a registered trademark of SonyComputer Entertainment, Inc.). The Cell BE architecture refers to aprocessor architecture which includes a base processor element, such asa Power Architecture-based control processor (PPE), connected tomultiple additional processor elements also referred to as SynergeticProcessing Elements (SPEs) and implementing a set of DMA commands forefficient communications between processor elements. In particular, SPEsmay be designed to handle certain types of processing tasks moreefficiently than others. For example, SPEs may be designed to moreefficiently handle processing video streams to identify and map thepoints of moving and stationary objects within a stream of frames. Inaddition, video processor 316, video processor 318, and sensor processor319 may implement other types of processor architecture that enablesefficient processing of video images to identify, in three-dimensions,moving and stationary objects within video images from which behavior ofconsumers and the stimulus to which a consumer responds, can bepredicted.

In the example, video processor 316, video processor 318, and sensorprocessor 319 each create and stream the properties, includingpositions, color, size, shape, and orientation, of the detected objectsto a geometry processor 320. In one example, each processed framestreamed to geometry processor 320 may include, but is not limited to, acamera ID, a frame number, a time stamp, and combinations of two or moreof X axis coordinates (x_loc), Y axis coordinates (y_loc), and Z axiscoordinates (z_loc). It is important to note that x_loc, y_loc, andz_loc may each include multiple sets of points and other data thatidentify all the properties of an object. If multiple objects aredetected and tracked within a single frame, the X axis coordinates and Yaxis coordinates for each object may be included in a single streamedobject property record or in multiple separate streamed object propertyrecords. In addition, a streamed property frame, such as the frame fromsensor processor 319 for a SONAR detected position, may include Z axislocation coordinates, listed as z_loc, for example.

Geometry processor 320 receives the 2D streamed object properties fromvideo processor 316 and video processor 318 and the other object datafrom sensor processor 319. Geometry processor 320 matches up thestreamed 2D object properties and other data and constructs 3D objectproperties 110 and consumer interface record 114 from the streamed 2Dobject properties and other data. In particular, geometry processor 320constructs 3D object properties 110 that include the depth of an object.In one example, each 3D object property record constructed by geometryprocessor 320 may include a time stamp, an object or consumer movementlabel, X axis coordinates (x_loc), Y axis coordinates (y_loc), and Zaxis coordinates (z_loc), and additional information collected fromsensors. For example, additional information collected from sensors mayinclude a location identifier received from an RFID or GPS detectedlocation coordinates, weather information, or other sensed informationabout a consumer environment. In addition, in particular, geometryprocessor 320 constructs consumer interface record 114 which may includea time stamp, a content type, and other information collected about theinterface or underlying system to controlling output at an interfaceaccessible to a consumer.

At any of video processor 316, video processor 318, sensor processor319, and geometry processor 320 property records may include at leastone identifier to enable persistence in tracking the object. Forexample, the identifier may include a unique identifier for the objectitself and an identifier of a class or type of object, including anobject identified as consumer movement.

In particular, by video processor 316, video processor 318, and sensorprocessor 319 identifying and classifying object properties, each of theprocessors may access definition database 112 for accessing previouslyprocessed inputs and behavior mappings to more accurately identify andclassify 2D object properties to detect and match the streamed 2D objectproperties to an object or behavior. In addition, geometry processor 320may more accurately construct 3D properties of objects based on thestreamed 2D object properties, based on previously matched andconstructed 3D properties of objects accessed from definition database112. Further, definition database 112 may store the streamed 2D objectproperties and 3D object properties for future reference.

In addition, by video processor 316, video processor 318, and sensorprocessor 319 identifying and classifying object properties and bygeometry processor constructing 3D object properties 110, each of theprocessors may identify detected objects, including behaviors ofmonitored users. For example, video processor 316, video processors 318,sensor processor 319, and geometry processor 320 may access definitiondatabase 112, which includes specifications for use in mapping facialexpressions and other body movements, performing facial and other bodymovement recognition, and performing additional processing to identifyan object representing a behavior by a consumer. In addition, videoprocessor 316, video processor 318, sensor processor 319 and geometryprocessor 320 may access definition database 112, which includesspecifications for types of products available within a consumerenvironment. Further, video processor 316, video processors 318, sensorprocessor 319, and geometry processor 320 may access definition database112, which includes specifications for different types of environmentsfor use in identifying a particular environment in which a consumer islocated based on detected objects and background. Additionally, inconstructing 3D object properties 110, video processor 316, videoprocessors 318, sensor processor 319, and geometry processor 320 mayidentify multiple detected objects in the environment and thereforeidentify multiple behaviors of a single consumer, one or moreinteractions between multiple consumers, or one or more behaviors by aconsumer responsive to one or more stimulus. By monitoring andidentifying interactions to a stimulus detected in the environment inwhich the stimulus is located, more accurate prediction of a behavior inthe context in which the behavior is made may be performed.

Referring now to FIG. 4, a block diagram illustrates one embodiment of aresponse interpreter system. It is important to note that the multiplecomponents depicted within response interpreter 106 may be incorporatedwithin a single system or distributed via a network across multiplesystems.

In the example, a 3D properties record 402 includes “time stamp”,“x_loc”, “y_loc”, and “z_loc” data elements. It will be understood that3D properties record 402 may include additional or alternate dataelements as determined by geometry processor 320 of FIG. 3. For example,3D properties record 402 may include additional information identifyinga particular or relative location of a consumer within a sensor area andnot just within the focus area, colors, and other data collected byimage capture devices and sensors and processed within 3D objectdetector 104.

In addition, in the example, a consumer interface record 400 includes“time stamp”, “interface identifier (ID)”, “content type”, and “displaymode”. In the example, an interface ID may include an identifier for aparticular interface within a consumer environment or a type ofinterface, the content type may indicate the specific content, aposition within a sequence of content, or other types of contentindicators, and a display mode may indicate which mode a consumer viewsthe content within, when multiple modes are available. It will beunderstood that additional or alternate types of data may be includedwithin consumer interface record 400

Response interpreter 106 includes a response interpreter controller 404,where response interpreter controller 404 may include one or moreprocessors programmed to interpret a behavior by a consumer responsiveto at least one stimulus within a consumer environment. For example,response interpreter controller 404 may include a processor with theCellBE architecture, programmed to efficiently process 3D objectproperties data streams and consumer interface record streams, predictbehaviors of consumers from the 3D object properties streams, andpredict what product the consumer behavior is in response to within aconsumer environment. In addition, response interpreter controller 404may include processors upon which software runs, where the softwaredirects processing of 3D object properties streams, predicts behaviorsof consumers from the 3D object properties streams, and predicts whatstimulus within the consumer environment the consumer behavior isresponsive to.

In processing 3D object properties streams, consumer interface records,predicting behaviors, and predicting what stimulus the consumer behavioris in response to within a consumer environment, response interpretercontroller 404 maps 3D object properties to one or more behavior actionswith a percentage probability that the streamed 3D object propertiesrepresent the mapped behavior actions and with a percentage probabilitythat the predicted behavior is in response to a particular productwithin a consumer environment. In particular, response interpretercontroller 404 accesses one or more behavior definitions for one or morebehaviors and determines whether the 3D object properties match one ormore characteristics of one or more behaviors as defined in one or moreof the behavior definitions. In addition, response interpretercontroller 404 accesses one or more product definitions for one or moreproducts and determines whether the 3D object properties match one ormore characteristics of one or more products as defined in one or moreof the product definitions. Behavior definitions may include mapped 3Dmodels of one or more types of behaviors. In addition, behaviordefinitions may define the parameters of identifying characteristics ofa behavior including, but not limited to, body part detected, type ofmovement, surface characteristics, shape, speed of movement, frequency,span of movement, depth of movement, temperature, and color.

In addition, behavior definitions are specified to enable responseinterpreter controller 404 to determine the overall response by aconsumer based on the predicted behaviors. For example, once responseinterpreter controller 404 determines that an object stream represents aconsumer behavior of “pulling eyebrows together” and the consumer isviewing an interface displaying a survey question to the consumer whilethe consumer is checking out, response interpreter controller 404 isenabled to determine, from definition database 112 that the consumer isconfused by the question displayed within the interface. In contrast, ifresponse interpreter controller 404 determines that the object streamrepresents the consumer behavior of “thrusting jaw forward” and theconsumer is viewing the same interface displaying a survey question tothe consumer, response interpreter controller 404 is enabled todetermine, from definition database 112 that the consumer is angered bythe survey question or the presence of the survey question whilechecking out. Thus, behavior and product definitions may predict aparticular response based on the combination of behaviors and theproduct to which a consumer appears to respond.

It is important to note that in interpreting 3D object propertiesstreams, response interpreter controller 404 performs an aggregateanalysis of all the tracked objects in one or more 3D object propertiesstreams identified for a particular focus area by one or more behaviorprocessing systems. In one example, response interpreter controller 404aggregates the 3D object property streams for a particular focus areaand particular sensor area. In another example, response interpretercontroller 404 may receive multiple 3D object properties streams fromareas overlapping a focus area and sensor area, analyze the 3D objectproperties streams for similarities, location indicators, andorientation indicators, and construct the 3D object properties streamsinto a 3D aggregate representation of an area.

In one embodiment, response interpreter controller 404 may map theaggregate of the tracked objects directly into a single behaviordefinition or a single product definition. In another embodiment,response interpreter controller 404 maps multiple aggregated trackedobjects into multiple behavior definitions or product definitions. Forexample, a person may simultaneously communicate through facial behaviorand a hand behavior, where in predicting the actual behaviorscommunicated through the tracked movement of the facial behavior andhand behavior, response interpreter controller 404 analyzes the 3Dobject properties of the facial behavior in correlation with the 3Dobject properties of the hand behavior and accesses behavior definitionsto enable prediction of each of the behaviors and to enable predictionof whether the behaviors, in combination, represent a particularpredicted response to a particular product. In another example, responseinterpreter controller 404 may aggregate and analyze multiple behaviorsresponse to multiple products to detect a pattern of a consumer stoppingto feel or pick up a particular brand, type, or composition of product.

In the example, response interpreter controller 404 accesses behaviordefinitions and product definitions from definition database 112, whichincludes general behavior definitions 412, store object definitions 414,store specific behavior definitions 416, consumer specific behaviordefinitions 418, and voice stress definitions 420. It will be understoodthat definition database 112 may include additional or alternate typesof behavior, object, and voice definitions. In addition, it is importantto note that each of the groupings of behavior, object, and voicedefinitions illustrated in the example may reside in a single databaseor may be accessed from multiple database and data storage systems via anetwork.

General behavior definitions 412 include behavior definitions for commonbehaviors and factors for determining what type of response a behavioris typically representative of within a consumer environment. Forexample, general behavior definitions 412 may include behaviorsdefinitions for common behaviors, such as a person pointing, a personwaving, a person nodding “yes” or shaking one's head “no”, or othertypes of common behaviors that are generally associated with particulartypes of responses within a consumer environment, but could representother types of responses depending on the context. For example, within ageneral consumer environment, a person shaking one's head “no” generallyindicates a response of “do not want”, however, additional factors suchas the speed at which the person shakes and whether the person'seyebrows are raised are also detected and effect the levels of detectioneffect whether the person's head shake could indicate a different typeof response to a product, such as a lack of understanding of a product.

Store object definitions 414 include object definitions for products andother elements within a particular consumer environment. For example,store object definitions 414 may include the object definitions for eachtype of product and service within a consumer environment such thatresponse interpreter 404 may more accurately identify products andservices within a particular store environment from 3D object propertystreams.

Store specific behavior definitions 416 include behavior definitionsspecific to the store environment or response-enabled application towhich predicted response output 108 will be sent. For example, in thepresent embodiment, predicted response output 108 may be output to aresponse-enabled application incorporated into a store attendantnotification system that provides audio and other types of notificationto store attendants to aid consumers, wherein store specific behaviordefinitions 416 includes selected behavior definitions focused on thetypes of behaviors as responses to products that would be relevant forthe store attendant notification system. In another example, in thepresent embodiment, predicted response output 108 may be output to aresponse-enabled application incorporated into a consumer interfacecontroller that adjusts the display of content within the interface,wherein store specific behavior definitions 416 includes selectedbehavior definitions focused on the types of behaviors as responses tostimulus that would be relevant for determining whether to adjustcontent based on consumer behavior.

In one example, in detecting consumer responses within a restaurant forcontrolling notifications to wait staff within the restaurant, storespecific behavior definitions 416 may specify types of behaviorsindicative of a consumer ready and waiting for a wait staff. Forexample, store specific behavior definitions 416 may include definitionsto enable response interpreter controller 404 to detect, for a consumersitting at a table, a number of glances by the consumer around arestaurant, indicators of a hand motion signaling the consumer is readyto order, or an amount of time expired since a consumer stopped lookingat a menu, where each of these behaviors separately or in combinationmay indicate a consumer waiting for a wait staff to come to the table.

In another example, in detecting consumer responses within a departmentstore for controlling notifications to store attendants to adjustproduct displays or suggest a product to a consumer, for example, storespecific behavior definitions 416 may specify types of behaviorsassociated with products within the store triggering response by a storeclerk. For example, in addition to identifying a particular product froman object definition in store object definitions 414, store specificdefinitions 416 may include definitions to enable response interpretercontroller 404 to detect a number of glances by a consumer towards aproduct before the consumer picks up the product or a number of times aconsumer picks up a product before the consumer leaves the product orplaces the product in a basket. In another example, based on storespecific definitions 416, response interpreter controller 404 may detecta number of times and a length of time a consumer looks at a price tagor other textual information affixed to a product and any behaviorsindicative of a consumer's response to the price, such as hesitation ordropping the product. In yet another example, based on store specificdefinitions 416, response interpreter controller 404 may detect whethera consumer breaks a product, attempts to remove a portion of a productor behaves in a manner indicative that the consumer may attempt to takethe product without payment. By tracking consumer hesitation, reading ofproduct information, revisiting products or areas of a store, irregularbehavior indicative of breaking or stealing and other responses by aconsumer to one or more products in a store, consumer responses aregathered and can be provided to a store attendant as a summary of aconsumer's activity or in an alert to a store clerk to attend to aconsumer or clean a mess. Additionally, consumer responses to a productmay be gathered to evaluate whether consumer responses match a predictedresponse and whether consumers understand how to use a product.

Consumer specific behavior definitions 418 include behavior definitionsspecific to a consumer. In one example, response interpreter controller404 accesses an identifier for a consumer from the consumer presentingscannable identification, such as from a unique consumer login code or aconsumer credit card. In another example, response interpretercontroller 404 accesses an identifier for a consumer from comparing anattribute of the consumer detected by 3D object detector 104, with adatabase of user attributes. For example, response interpretercontroller 404 may perform facial or voice recognition. It will beunderstood that response interpreter controller 404 may perform othertypes of identity access and authentication of a consumer. [0091]Further, within the available definitions, a definition may beassociated with a particular area of movement or a particular depth ofmovement within a 3D focus area. In particular, the three-dimensionalfocus area in which movement is detected may be divided intothree-dimensional portions, where movements made in each of the portionsmay be interpreted under different selections of behavior definitions.For example, one three-dimensional portion of a focus area may beconsidered an “active region” where movement detected within the area iscompared with a selection of definitions associated with that particularactive region, such as a region within a particular distance of acheckout line, a kiosk, a product display, or other points within aconsumer environment.

Additionally, definition database 112 may include voice stressdefinitions 430. Voice stress definitions 430 include definitions forlevels of voice characteristics including, but not limited to, volume,clarity, and speed, where particular levels of voice characteristics areassociated with behaviors. For example, an elevated, rapid voicecharacteristic may indicate or correlate with behaviors indicating aconsumer is frustrated or angry in response to a particular productwithin a consumer environment. By combining detected voice samples of aconsumer, as gathered by sensor 240, for example, and comparing thevoice samples with voice stress definitions 430, response interpretercontroller 404 may more accurately predict behaviors from 3D objectstreams where matching voice stress definitions add to the predictedprobability and therefore more accurately predict a response by aconsumer to a stimulus within a consumer environment.

The behavior, object, and voice stress definitions included withindefinition database 112 may be added to or adjusted based on userfeedback of a consumer, store attendant, or other user. For example,definition database 112 may learn additional behavior, object, and voicestress definitions and adjust the parameters of already learnedbehavior, object, and voice stress definitions through a store attendantresponding to a consumer based on predicted response record 108 and thestore attendant indicating whether the predicted consumer response iscorrect, in a similar manner as a speech recognition system is trained,to more accurately map and predict behaviors, objects, and voice stressin general, within different context, specific to applications, andspecific to particular consumers and particular consumer environments.

Response interpreter controller 404 may output predicted response output108 in the form of one or more response records, such as response record420. Response record 420 indicates at least one “response type”,“predicted behaviors”, “probability % as each behavior”, and “stimulus”.In addition, response record 420 includes the start X, Y, and Z axisproperties and ending X, Y, and Z axis properties of the each detectedbehavior and stimulus, indicative of the location, direction ofmovement, and speed of movement of the behavior and stimulus, listed as“start_x_pos”, “end_x_pos”, “start_y_pos”, “end_y_pos”, “start z pos”,“end z pos”. Although not depicted, response record 420 may alsoindicate the X, Y, and Z axis start and ending properties of generalconsumer movement, indicative of the location, direction of movement,and speed of movement of the consumer. Further, additional informationacquired from sensors, such as RFID data, GPS coordinates, skin surfacecharacteristics, weather conditions, and other sensed data, may beassociated with a particular behavior record or included in a separateobject record.

In passing response record 420, response interpreter controller 404 mayfilter out particular types of response records. For example, responseinterpreter controller 404 may determine which types of responses shouldbe passed to a particular response-enabled application.

With reference now to FIG. 5, a block diagram depicts one embodiment ofa computing system in which the present invention may be implemented.The controllers and systems of the present invention may be executed ina variety of systems, including a variety of computing systems, such ascomputer system 500, communicatively connected to a network, such asnetwork 502.

Computer system 500 includes a bus 522 or other communication device forcommunicating information within computer system 500, and at least oneprocessing device such as processor 512, coupled to bus 522 forprocessing information. Bus 522 preferably includes low-latency andhigher latency paths that are connected by bridges and adapters andcontrolled within computer system 500 by multiple bus controllers. Whenimplemented as a server, computer system 500 may include multipleprocessors designed to improve network servicing power. Where multipleprocessors share bus 522, an additional controller (not depicted) formanaging bus access and locks may be implemented.

Processor 512 may be a general-purpose processor such as IBM's PowerPC™processor that, during normal operation, processes data under thecontrol of an operating system 560, application software 570, middleware(not depicted), and other code accessible from a dynamic storage devicesuch as random access memory (RAM) 514, a static storage device such asRead Only Memory (ROM) 516, a data storage device, such as mass storagedevice 518, or other data storage medium. In one example, processor 512may further implement the CellBE architecture to more efficientlyprocess complex streams of data in 3D. It will be understood thatprocessor 512 may implement other types of processor architectures. Inaddition, it is important to note that processor 512 may representmultiple processor chips connected locally or through a network andenabled to efficiently distribute processing tasks.

In one embodiment, the operations performed by processor 512 may control3D behavior detection from captured images and data for a consumerenvironment, behavior prediction from the detected behaviors, andcontrol of output to store attendants or systems within a consumerenvironment in the operations of flowcharts of FIGS. 12-15 and otheroperations described herein. Operations performed by processor 512 maybe requested by operating system 560, application software 570,middleware or other code or the steps of the present invention might beperformed by specific hardware components that contain hardwired logicfor performing the steps, or by any combination of programmed computercomponents and custom hardware components.

The present invention may be provided as a computer program product,included on a computer or machine-readable medium having stored thereonthe executable instructions of a computer-readable program that whenexecuted on computer system 500 cause computer system 500 to perform aprocess according to the present invention. The terms “computer-readablemedium” or “machine-readable medium” as used herein includes any mediumthat participates in providing instructions to processor 512 or othercomponents of computer system 500 for execution. Such a medium may takemany forms including, but not limited to, storage type media, such asnon-volatile media and volatile media, and transmission media. Commonforms of non-volatile media include, for example, a floppy disk, aflexible disk, a hard disk, magnetic tape or any other magnetic medium,a compact disc ROM (CD-ROM) or any other optical medium, punch cards orany other physical medium with patterns of holes, a programmable ROM(PROM), an erasable PROM (EPROM), electrically EPROM (EEPROM), a flashmemory, any other memory chip or cartridge, or any other medium fromwhich computer system 500 can read and which is suitable for storinginstructions. In the present embodiment, an example of a non-volatilemedium is mass storage device 518 which as depicted is an internalcomponent of computer system 500, but will be understood to also beprovided by an external device. Volatile media include dynamic memorysuch as RAM 514. Transmission media include coaxial cables, copper wireor fiber optics, including the wires that comprise bus 522. Transmissionmedia can also take the form of acoustic or light waves, such as thosegenerated during radio frequency or infrared data communications.

Moreover, the present invention may be downloaded or distributed as acomputer program product, wherein the computer-readable programinstructions may be transmitted from a remote computer such as a server540 to requesting computer system 500 by way of data signals embodied ina carrier wave or other propagation medium via network 502 to a networklink 534 (e.g. a modem or network connection) to a communicationsinterface 532 coupled to bus 522. In one example, where processor 512includes multiple processor elements, then a processing task distributedamong the processor elements, whether locally or via a network, mayrepresent a computer program product, where the processing task includesprogram instructions for performing a process or program instructionsfor accessing Java (Java is a registered trademark of Sun Microsystems,Inc.) objects or other executables for performing a process.Communications interface 532 provides a two-way data communicationscoupling to network link 534 that may be connected, for example, to alocal area network (LAN), wide area network (WAN), or directly to anInternet Service Provider (ISP). In particular, network link 534 mayprovide wired and/or wireless network communications to one or morenetworks, such as network 502. Further, although not depicted,communication interface 532 may include software, such as devicedrivers, hardware, such as adapters, and other controllers that enablecommunication. When implemented as a server, computer system 500 mayinclude multiple communication interfaces accessible via multipleperipheral component interconnect (PCI) bus bridges connected to aninput/output controller, for example. In this manner, computer system500 allows connections to multiple clients via multiple separate portsand each port may also support multiple connections to multiple clients.

Network link 534 and network 502 both use electrical, electromagnetic,or optical signals that carry digital data streams. The signals throughthe various networks and the signals on network link 534 and throughcommunication interface 532, which carry the digital data to and fromcomputer system 500, may be forms of carrier waves transporting theinformation.

In addition, computer system 500 may include multiple peripheralcomponents that facilitate input and output. These peripheral componentsare connected to multiple controllers, adapters, and expansion slots,such as input/output (I/O) interface 526, coupled to one of the multiplelevels of bus 522. For example, input device 524 may include, forexample, a microphone, a video capture device, a body scanning system, akeyboard, a mouse, or other input peripheral device, communicativelyenabled on bus 522 via I/O interface 526 controlling inputs. Inaddition, for example, an output device 520 communicatively enabled onbus 522 via I/O interface 526 for controlling outputs may include, forexample, one or more graphical display devices, audio speakers, andtactile detectable output interfaces, but may also include other outputinterfaces. In alternate embodiments of the present invention,additional or alternate input and output peripheral components may beadded.

Those of ordinary skill in the art will appreciate that the hardwaredepicted in FIG. 5 may vary. Furthermore, those of ordinary skill in theart will appreciate that the depicted example is not meant to implyarchitectural limitations with respect to the present invention.

Referring now to FIG. 6, a block diagram depicts one example of adistributed network environment in which a response-enabled method,system, and program may be implemented. It is important to note thatdistributed network environment 600 is illustrative of one type ofnetwork environment in which the response-enabled method, system, andprogram product may be implemented, however, the response-enabledmethod, system, and program product may be implemented in other networkenvironments. In addition, it is important to note that the distributionof systems within distributed network environment 600 is illustrative ofa distribution of systems; however, other distributions of systemswithin a network environment may be implemented. Further, it isimportant to note that, in the example, the systems depicted arerepresentative of the types of systems and services that may be accessedor request access in implementing a response processing system and aresponse-enabled system. It will be understood that other types ofsystems and services and other groupings of systems and services in anetwork environment may implement the response processing system andresponse-enabled system.

As illustrated, multiple systems within distributed network environment600 may be communicatively connected via network 502, which is themedium used to provide communications links between various devices andcomputer communicatively connected. Network 502 may include permanentconnections such as wire or fiber optics cables and temporaryconnections made through telephone connections and wireless transmissionconnections, for example. Network 502 may represent bothpacket-switching based and telephony based networks, local area and widearea networks, public and private networks. It will be understood thatFIG. 6 is representative of one example of a distributed communicationnetwork for supporting a response processing system and response-enabledsystem; however other network configurations and network components maybe implemented.

The network environment depicted in FIG. 6 may implement multiple typesof network architectures. In one example, the network environment may beimplemented using a client/server architecture, where computing systemsrequesting data or processes are referred to as clients and computingsystems processing data requests and processes are referred to asservers. It will be understood that a client system may perform as botha client and server and a server system may perform as both a client anda server, within a client/server architecture. In addition, it will beunderstood that other types of network architectures and combinations ofnetwork architectures may be implemented.

In the example, distributed network environment 600 includes a clientsystem 602 with an image capture system 604 and a client system 606 withan image capture system 608. In one example, image capture systems 604and 608 are stereoscopic image devices implementing one or more imagecapture devices, such as image capture devices 202 and 204, and mayinclude one or more sensors, such as sensor 240. Image capture systems604 and 608 capture images and other data and stream the images andother data to other systems via network 502 for processing. In addition,image capture systems 604 and 608 may include video processors fortracking object properties, such as video processor 316 and videoprocessor 318, described with reference to FIG. 3 and a geometryprocessor for generating streams of 3D object properties, such asgeometry processor 320, described with reference to FIG. 3.

In one example, each of client system 602 and client system 606 maystream captured image frames to one or more object detection services.In one example, a response processing service provider server 620provides a service that includes both an object detector service, suchas 3D object detector 104, for processing streamed images and other dataand a response interpreter service, such as response interpreter 106,for predicting a type of behavior, predicting a probability that thecaptured images represent the predicted type of behavior, andpredicting, based on the behaviors, a response by the consumer to aparticular stimulus within the consumer environment, and controllingoutput of the predicted response records to one or more other systemsaccessible via network 502.

As to response processing service provider server 620, differententities may implement a response processing service and differententities may access the response processing service. In one example, auser logged into one of client systems 602 or client system 606 maysubscribe to the response processing service. In another example, animage capture system or a particular application requesting responseprocessing may automatically stream captured images and data to theresponse processing service. In yet another example, a business or otherentity may implement the response processing service in a communicationsnetwork.

In another example, each of client system 602 and client system 606 maystream captured frames to a 3D object detector server 624. 3D objectdetector server 624 receives captured images and other data from imagecapture systems, such as image capture system 604 or image capturesystem 608, and processes the images and other data to generate 3Dobject properties of detected behaviors, for output to a responseinterpreter system, such as response interpreter server 622 or responseprocessing service provider server 620. In additional or alternateembodiments, an object detector service may be implemented within one ormore other systems, with one or more other services performed withinthose systems. In particular, in additional or alternate embodiments, anobject detector service may be implemented within a client system atwhich the images and other data are captured.

Each of the server systems described may be distributed across one ormore systems. In addition, each of the server systems may be distributedacross systems with 3D image processing power, including processors withthe CellBE architecture programmed to perform efficient 3D dataprocessing. In one example, an entity, such as a business or serviceprovider, may implement separate server systems for object detection andbehavior interpretation, wherein multiple response interpreter serversare implemented with each response interpreter server processingdifferent types of 3D object properties.

Response processing service provider server 620, response interpreterserver 622, and 3D object detector server 624 may locally store adefinition database, such as definition database 112, of raw images, 3Dbehavior properties, behavior definitions, object definitions, and voicestress definitions, for example. In addition, response processingservice provider server 620, response interpreter server 622 and 3Dobject detector server 624 may access a definition database serviceserver 626 that facilitates definition database 112. As previouslydescribed, definition database 112 may include, but is not limited to,raw images and data, 3D behavior properties, behavior definitions,object definitions, voice stress definitions and object predictions.

In addition, definition database service server 626 includes a behaviorlearning controller 630. Behavior learning controller 630 prompts usersto provide samples of particular types of behaviors responsive to aparticular stimulus and prompts users to indicate whether a predictedtype of behavior matches an actual behavior by the consumer to theparticular stimulus. In addition, behavior learning controller 630gathers other information that enables behavior learning controller 630to learn and maintain behavior information in definition database 112that when accessed by object detector services and response interpreterservices, increases the accuracy of generation of 3D object properties,the accuracy of prediction of behaviors, and the accuracy of predictionof a consumer response to a stimulus based on the behaviors from 3Dobject properties by these services.

Further, response processing service provider server 620, responseinterpreter server 622, 3D object detector server 624, or definitiondatabase service server 626 may access additional context informationfor a consumer, specifying behavior definitions for the consumer, from aclient profile service server 640. In addition, client profile serviceserver 640 may provide definitions particular to a store or consumerenvironment. In addition, client profile service server 640 may monitorand provide additional information about a location of a consumer frommonitored information such as the current location of the consumer, thecurrent physical environment in which the consumer is located, and theevents currently scheduled for a consumer. In one example, clientprofile service provider 640 monitors a consumer's electronic calendaror a consumer's current GPS location, for example, from the consumer'spersonal, portable telephony device.

Response processing service provider server 620 and response interpreterserver 622 stream predicted response records, such as response records420, to response-enabled applications via network 502. In the exampleembodiment, client system 606 includes a consumer interaction system 610which is a response-enabled application, enabling client system 606 todetermine and control signals such as outputs to a store attendantsystem or an interface within a consumer environment. Consumerinteraction system 610 at client system 606 may receive predictedresponse records from client system 606, as captured by image capturesystem 608, or may receive predicted response records based on imagesand data detected by image capture system 608 or other image capturesystems.

In addition, in the example embodiment, client service provider server612 includes a consumer interaction system 614 which is aresponse-enabled service for enabling client service provider server 612to determine and output control signals based on response records andother data. Client service provider server 612 represents a server thatprovides a consumer interaction service to one or more subscribingclient systems. Consumer interaction system 614 may receive responserecords for a particular consumer environment from one or more systemsand determine and output control signals to one or more client systems.

With reference now to FIG. 7, a block diagram illustrates one example ofan implementation of a response interpreter system communicating with aresponse-enabled consumer interaction system. Consumer interactionsystem 700 receives predicted behavior records from response processingsystem 100, which enables consumer interaction system 700 to providecontrol signals based on real-time, 3D images and other data capturedwithin a consumer environment to adjust a consumer's experience of theconsumer environment.

In particular, consumer interaction controller 708 of consumerinteraction system 700 may base control signals on response records,such as response record 702. In addition, consumer interactioncontroller 708 may enhance or generate control signals based on otherdata, combined with the data included in response record 702. Forexample, consumer interaction controller 708 may access static ordynamic information about a consumer environment, through accessingfloor maps, display maps, web site maps, photographs, video streams,RFID information, weather information and other data about a monitoredconsumer environment. In addition, consumer interaction controller 708may access information about a consumer within a consumer environment,through other imaging devices. Then, consumer interaction controller 708may generate a mapping of the predicted response records at locationswithin the monitored consumer environment.

In particular, in the example, consumer interaction system 700 receivesresponse record 702 and consumer interaction controller 708 translatesresponse record 702 into inputs that facilitate modeling of anenvironment and generation of control signals for potential adjustmentof a consumer's experience within a consumer environment. In theembodiment depicted, in translating response record 702, consumerinteraction controller 708 accesses response translation database 710 totranslate one or more predicted responses, based on predicted behaviors,to a stimulus, into control signals based on the type of device to whichthe control signals will be output. In other embodiments, responserecords may be translated into other types of data usable by consumerinteraction controller 708 for implementing a consumer interactionsystem. A preferences database 706 may specify additional preferencesfor selecting control signals for a particular consumer or a particularconsumer environment. In addition, response translation database 710 maybe associated with a particular consumer or particular consumerenvironment selected according to other criteria from among a selectionof multiple response translation databases available to consumerinteraction system 700.

In the example, response translation database 710 includes, for eachentry, data specified in multiple categories. In particular, in theexample, each entry includes a response type, a selection of predictedbehaviors upon which the response type is based, a minimum (min)percentage probability that the response is accurately predicted, astimulus responded to by the consumer. In addition, each entry includesinstructions for generating control signals, if applicable, to each of astore attendant notification system, an output interface accessible tothe consumer, or to other systems accessible to a consumer within aconsumer environment. It will be understood that additional or alternatetypes of data may be included in response translation database 710 fortranslating response records into control signals.

In particular, in the example, a response type included in responserecord 702 includes a predicted response of “confused” by a consumer toa stimulus “page 4 of registry” with a 90% probability that thebehaviors, which combined indicate the predicted response, areaccurately predicted. Response record 702 is illustrative of, but notlimiting of, the types of data and detail of data that may be includedin a response record output by response processing system 100 andreceived by consumer interaction system 700. In one example, responserecord 702 may also include a percentage probability that the predictedtype of response represents a consumer problem. In another example,response record 702 may only include the predicted behaviors andconsumer interaction controller 708 then either predicts a response typeor translates behaviors into control signals.

In the example, consumer interaction controller 708 may control outputof control signals to one or more of the devices illustrated. For theexample, consumer interaction controller 708 compares response record702 with entry 720 of response translation database 710 and may generatecontrol signals to one or more of a registry program controller 730 anda store attendant audio feedback controller 760 based on control signalsgenerated from comparing response record 702 and other collected datawith entry 720. It will be understood that consumer interactioncontroller 708 may output control signals to additional or alternatetypes of output devices and to multiple of a same type of output device.

In controlling control signals based on response record 702, in oneexample, consumer interaction controller 708 may determine that acontrol signal needs to be sent to trigger a novice mode, as illustratedat reference numeral 732, to registry program controller 730. Forexample, a registry program running at a kiosk may provide multiplemodes of operation, such as a novice or beginner mode, a general mode,and an expert mode. Depending on the mode, registry program controller730 adjusts the amount and detail of instructions displayed to theconsumer for interacting with the registry kiosk service provided withina consumer environment, whether on-line or within a store. By switchingto a novice mode, when the detected 3D image stream of a consumeraccessing a registry program is indicated as “confused”, the consumer'sexperience within the consumer environment is adjusted in an attempt toprovide the consumer with additional assistance in using an electronicuser interface.

Although not depicted, other examples of control signals for controllinga program for an output interface may include instructions tographically adjust a portion of displayed content to highlightparticular content, instructions to change the images of products orother images on an output interface, or instructions to adjust the typesof outputs at an output interface, such as adding or removing an audiooutput signal.

In addition, in controlling control signals based on response record702, in one example, consumer interaction controller 708 may determinethat a control signal needs to be sent to trigger an audio message toone or more store attendants to prompt store attendants to provideadditional assistance to the consumer using the registry service, asillustrated at reference numeral 762. If on-line, an on-line storeattendant instant communication interface may be triggered and a storeattendant assigned to communicate with the consumer via the instancemessage communication interface. If within a store environment, in theexample, store attendant audio feedback controller 760 receives controlsignal 762 to assist the consumer at the registry terminal and storeattendant audio feedback controller 760 determines a specific one ormore store attendants wearing earpieces to which the message to assistthe consumer at the registry terminal is output. In addition tocontrolling output of a message to assist a consumer, the control signalmay also indicate the consumer's behavior or response which triggeredthe control signal. For example, if a consumer response indicates aconsumer interest in a particular brand or material because the consumerhas picked up multiple products of a same brand or material throughout astore, the control signal may prompt the store attendant with theconsumer preferred brand or material.

In addition to entry 720, entries 722, 724, and 726 of responsetranslation database 710 are further illustrative of types of entriesthat may be included in response translation database 710. For example,entry 722 specifies that for a response of “impatient” within minimumpredicted probability of 70% and to a stimulus of “registry kiosk”, if astore attendant is triggered, the store attendant is prompted to offerassistance to the consumer in locating products within the storeenvironment and if the registry program is triggered, the registryprogram controller is directed to skip any pages within a sequence ofpages that are only product advertisements.

Entries 724 and 726 are illustrative of entries that may trigger acontrol signal to another system other than a system with which theconsumer is directly interacting. For example, entry 724 specifies thatfor a response of “impatient” within minimum predicted probability of80% and a stimulus of “waiting in line for a registry kiosk”, if a storeattendant is triggered, the store attendant is prompted to check on thelines for the registry kiosk and if other systems are triggered, anysystems within a store which could be implemented as a registry system,are triggered to operate as a registry system. For example, a registerwithin a store environment may operate in one mode as a registry systemand in another mode as a cash register, where consumer interactioncontroller 708 may generate a control signal for adjusting the operatingmode and alerting a consumer to the change in mode. In addition, in theexample, entry 726 specifies that for a response of “upset” with minimumpredicted probability of 70% and a stimulus of “waiting in the check-outline”, if a store attendant is triggered, the store attendant isprompted to send an additional employee to open an additional register,and if other systems are triggered, one or more self-checkout systemsare triggered to open. In some store environments, a store attendantmonitors a particular number of self-checkout systems to provideassistance to consumers, but self-checkout systems may be closed ifthere is not sufficient consumer use to afford positioning a storeattendant to monitor the self-checkout systems. Consumer interactioncontroller 708, however, may send a control signal to open one or moreclosed self-checkout systems based on the current behaviors of consumerswaiting in a checkout line to attempt to change a consumer's experienceof the consumer environment.

It is important to note that in one example, response processing system100 may automatically monitor one or more areas within a consumerenvironment or one or more online store servers and may automaticallytrigger prediction of consumer responses for each consumer enteringwithin one of the areas within the consumer environment or accessing theonline store interface. Examples of monitored areas within a consumerenvironment may include, but are not limited to, a selected areaincluding a point of sale terminal, a selected area including a displaywithin a store, and a selected area including a kiosk. In addition,consumer interaction system 700 may monitoring one or more areas withina consumer environment and trigger response processing system 100 tocapture 3D image streams of a consumer and predict a consumer responseof a consumer within the consumer environment.

With reference now to FIG. 8, a block diagram illustrates one example ofan object detection interface and a consumer interaction interfacewithin a store environment.

In the example, FIG. 8 includes image capture system 804, which, in oneexample, include one or more of image capture device 202, image capturedevice 204, and sensor 240. Image capture system 804 captures a 3D imagestream of the behaviors of a consumer within area 818, an audio streamof audio detectable within area 818, and data about the contentdisplayed within a display interface 810 of store kiosk 802.

In particular, in the example, area 818 represents a consumerenvironment in which a consumer interacts with display interface 810 ofstore kiosk 802. Store kiosk 802, in one embodiment, provides a serviceto a consumer within a store environment. For example, store kiosk 802may provide a registry service, a financial service, a check-outservice, or other service provided to consumers within the storeenvironment.

In the example, image capture system 804 and store kiosk 802 areillustrated in combination, separate from a computer system 812. In analternate embodiment, image capture system 804 may be implemented withinstore kiosk 802. In another embodiment, image capture system 804 maycapture 3D image streams, audio streams, and content display acrossmultiple kiosks or areas within a consumer environment. Additionally, inanother embodiment, computer system 812, image capture system 804, andstore kiosk 802 may be implemented within a single physical unit ordistributed across multiple systems communicatively connected by anetwork, such as network 502.

Computer system 812 includes a response processing system 100 and aconsumer interaction system 700. Thus, response processing system 100receives images and other data from image capture system 804 andgenerates a predicted response record. Consumer interaction system 700receives the predicted response record and generates control signals foroutput to store kiosk 802 or an audio feedback controller 820. Storekiosk 802 includes a program controller for receiving control signalsfrom consumer interaction system 700 and adjusting content displayedwithin display interface 810 to a consumer based on the control signals.Audio feedback controller 820 controls output of audio signals via alocal wireless network to headphones worn by store attendants, such asheadphones 822. It will be understood that response processing system100 and consumer interaction system 700 are illustrated within a singlecomputer system 812, computer system 812 may be distributed across anetwork of computer systems communicatively connected via network 502,for example.

In particular, more and more kiosks or other computer interfaces areprovided within a store environment. By coupling image capture system804 with store kiosk 802, the content displayed to a consumer withindisplay interface 810 is adjusted not only in response to the selectionsmade by the consumer, but also in an attempt to adjust a consumer'sexperience based on the behavior of the consumer indicative of aresponse to the consumer's interaction with store kiosk 802.

Additionally, in the example, response processing system 100 may trackand predict responses to stimulus based on the consumer's proximity tothe stimulus. In the example, response processing system 100 detects a3D image stream of area 818, however, response processing system 100tracks and interprets behaviors made by a consumer based on the sectionof area 818 in which the consumer behavior is made. For example, section830 is closer to display interface 810 and section 832 is further awayfrom display interface 810. A consumer behavior detected within section830 of area 818 is more likely responsive to the content displayedwithin display interface 810. In contrast, a consumer behavior detectedwithin section 832 of area 818 is more likely responsive to somestimulus other than the content displayed within display interface 810.

In one example, store kiosk 802 may provide a service to a consumer, butstore kiosk 802 may also control display of advertising or surveyquestions. Based on the consumer's detected behavior through a 3D imagestream captured by image capture system 804, store kiosk 802 may adjustthe advertising or survey questions displayed for a particular consumer,to remove the additional material to an impatient consumer, to addadditional material to a compliant consumer, or to change the materialfor a confused consumer.

In another example, as consumers move from store to store, consumers maybe required to interact with different kiosks at each store, however,the content displayed in kiosks from store to store may vary. While someconsumers may quickly adjust to and interact with different content anddifferent series of questions at different interfaces from store tostore, for many consumers, interacting with new and different contentand interface arrangements at different stores may cause frustration orconfusion. Thus, based on the consumer's detected behavior through a 3Dimage stream captured by image capture system 804, each store kiosk 802may adjust the complexity of content displayed within an interface, thearrangement of content within an interface, or otherwise change thecontent displayed within an interface in an attempt to change aconsumer's experience interacting with store kiosk 802.

Further, in another example, store kiosk 802 may only display content toa consumer, such as an entertainment display for showing productadvertisements, videos, or other in-store promotion. The consumer doesnot directly interact with store kiosk 802, however, image capturesystem 804 capture the consumer's response to viewing the contentdisplayed by store kiosk 802. In one example, if the consumer'sbehaviors indicate a response that the consumer is bored with thecontent displayed by store kiosk 802, store kiosk 802 may adjust thedisplayed content on display interface 810 in an attempt to change theconsumer's experience to one of entertained, rather than bored.

Referring now to FIG. 9, an illustrative diagram depicts one example ofa consumer interacting with an online consumer environment. In theexample, a consumer 902 interacts with an interface 904 of a computersystem 906 through which an online store or other consumer environmentis accessed via network 502 from one or more server systems, such asonline store server 910. Image capture system 804 captures a 3D imagestream of consumer 902 and other sensed data as consumer 902 responds tothe content displayed within interface 904 as delivered by a storecontroller 912. In the illustration, the consumer's behavior includesgritted teeth tensed eyebrows, and shaking fists.

The captured 3D image stream and other sensed data are transmitted toonline store server 910. Online store server 910 passes the 3D imagestream and other sensed data to response processing system 100. Responseprocessing system 100 determines a predicted response record. In theexample, a predicted response record may indicate, based on the detectedbehaviors, that the consumer is irritated by the displayed content forsome reason. Examples of stimulus that may cause a consumer irritationwhile viewing online store content include, but are not limited to, aconsumer continuing to receive error messages when trying to fill out aform, a consumer closing pop-up ads, video or music, or another actionwhere the consumer is impeded in moving to a next page within an onlinestore.

Consumer interaction system 700 generates control signals, based on thepredicted response record, for directing store controller 912 to adjustthe content delivered to computer system 906 for output via interface904 to consumer 902. While in the embodiment depicted, responseprocessing system 100 and consumer interaction system 700 are depictedas components of online store server 910, it is important to note one ormore of response processing system 100 and consumer interaction system700 may be provided as services to online store server 910 and that eachof the systems may be distributed across a network as illustrated inFIG. 6.

With reference now to FIG. 10, a block diagram illustrates a surveycontroller for aggregating consumer responses to a particular stimulus.In the example, the response-enabled application is a survey controller1010. Survey controller 1010 receives response records from responseprocessing systems, such as response record 702 from response processingsystem 100. Survey controller 1010 includes a user aggregate responsedatabase 1012 for storing received response records. In addition, surveycontroller 1010 includes a statistic calculator 1016 for calculating,from multiple consumer response records, statistics associated withconsumer responses to a particular stimulus within a consumerenvironment. Survey controller 1010 includes a product aggregateresponse database 1014 for storing the calculated statistics.

In one example, survey controller 1010 calculates statistics forconsumer responses to a particular stimulus within a consumerenvironment and transmits calculated statistics 1020 to consumerinteraction system 700. Consumer interaction system 700 may determinecontrol signals based on calculated statistics 1020. In addition,consumer interaction system 700 may adjust levels and requirementswithin response translation database 710 based on calculated statistics1020.

In other examples, survey controller 1010 may update other systems vianetwork 502 with calculated statistics. Further, survey controller 1010may process statistics results and generate control signals forcontrolling other systems, such as for controlling an application thatcontrols the content output to a consumer at a consumer interface.

In the example, aggregated responses are tracked for each display pageof a sequence of display pages implemented in a financial check-outsystem. In the example, for each of the pages, as illustrated atreference numerals 1002, 1004, and 1006, the responses of multipleconsumers to each page are recorded. By aggregating the responses ofmultiple consumers to a same page, statistic calculator 1016 can predicthow consumers in the future will respond to the same page, detect whenconsumer behavior indicates an acceptance of particular content, andcalculate other group response indicators.

Although not depicted, in another example, statistic calculator 1016 mayaggregate the responses of a single consumer to a same page on differentdays when the consumer interacts with the page. By monitoring a sameconsumer's response to a page, statistic calculator 1016 may monitor areaction by a consumer to the same content and predict when the contentneeds to be changed or identify how many times a consumer views the samecontent before showing behaviors that indicate understanding of thecontent.

Referring now to FIG. 11, a block diagram illustrates a response-enabledoperating system for adjusting content based on consumer behaviorresponsive to operations controlled by the operating system. In theexample, an operating system controller 1102 for a computer system 1100receives predicted response records from response processing system 100.In the example, response processing system 100 detects detectablebehavior movement 102 within a consumer environment through detecting a3D image stream and other data about a consumer. In addition, responseprocessing system 100 detects the contents of display interface 1110 towhich detected consumer behavior is responsive.

In the example, operating system controller 1102 receives predictedresponse records from response processing system 100 and generatescontrol signals for adjusting the performance of the operating system orapplications run by the operating system, based on entries in a responsetranslation database 1120. In the example, response translation database1120 includes a first entry 1104 for directing that if a response recordindicates a consumer behavior is “confused” and the consumer stimulusincludes a sequence of searching menus without making selections, then acontrol signal should be generated to trigger the operating system orapplication help function. In addition, in the example, responsetranslation database 1120 includes a second entry 1106 for directingthat if a response record indicates a consumer behavior is “distressed”and the consumer stimulus includes the user deleting words typed in thesame space within a document multiple times, then a control signalshould be generated to trigger the operating system or applicationthesaurus. It will be understood that in additional or alternate typesof entries within response translation database, a consumer's behaviorbased response coupled with consumer activity at a computer system maytrigger an operating system to perform additional functions or commandsof the operating system or an application.

By enabling operating system controller 1102 to respond to predictedresponse records from response processing system 100, a consumer'sdetected behaviors influence how operating system controller 1102controls a computer system. In addition to generating control signals tocontrol the operating system and application at a computer system,operating system controller 1102 may generate and send control signalsto a server system via a network, wherein the control signals direct theclient or server system to adjust the content delivered to the computersystem controlled by operating system controller 1102.

Referring now to FIG. 12, a high level logic flowchart depicts a processand program for a response processing system to predict response typesand stimulus responded to with a percentage probability. In the example,the process starts at block 1200, and thereafter proceeds to block 1202.Block 1202 depicts capturing, via a stereoscopic image device, multipleimage streams and via sensors, sensor data, within a focus area. Next,block 1204 illustrates tracking objects within the images and sensordata. Thereafter, block 1206 depicts generating a stream of 3D objectproperties for tracked objects and a stream of consumer interfacerecords for tracked content within an interface accessible to aconsumer. Thereafter, block 1208 depicts aggregating the 3D objectproperties for each of the tracked objects and the consumer interfacerecords. In particular, the aggregated 3D object properties representone or more objects tracked in association with at least one consumerrepresentative of behaviors of the at least one consumer and products orother detectable objects within the consumer environment.

Next, block 1210 illustrates predicting at least one type of behaviorfrom the aggregated stream of 3D object properties from one or morebehavior definitions that match the aggregated stream of 3D objectproperties with a percentage probability. In addition, next, block 1212depicts predicting at least one stimulus within the consumer environmentto which the predicted behaviors are responsive to from the collectedconsumer interface records and additional 3D object properties whichwhen compared with product definitions identify a product within theconsumer environment. Thereafter, block 1214 depicts transmitting eachoverall predicted response to the predicted stimulus based on thepredicted behaviors with the percentage probability that the predictedbehaviors are accurately predicted, and the process ends.

With reference now to FIG. 13, a high level logic flowchart depicts aprocess and program for response detection by tracking objects withinimage streams and other sensed data and generating 3D object propertiesfor the tracked objects representative of behaviors and generatingconsumer interface records indicative of any content accessible to theconsumer at an interface within the consumer environment. Asillustrated, the process starts at block 1300 and thereafter proceeds toblock 1302. Block 1302 depicts an object detector system receivingmultiple image streams, via one or more image capture devices, andsensed data, via one or more sensors. Next, block 1304 illustrates theobject detector system attaching metadata to the image frames and senseddata, and the process passes to block 1306. In one example, metadataincludes data such as, but not limited to, a camera identifier, framenumber, timestamp, and pixel count.

Block 1306 depicts the object detector system processing each imagestream and sensed data to detect and track objects, wherein objects mayinclude physical objects and consumer movement indicative of a behavior,and to detect and track content accessible to a consumer at an interfacewithin the consumer environment. Next, block 1308 illustrates generatingstreams of tracked object properties with metadata from each imagestream. Thereafter, block 1310 depicts combining the tracked objectproperties to generate 3D object properties with metadata and combiningthe tracked content accessible to a consumer to generate consumerinterface records. Next, block 1312 illustrates transmitting the 3Dtracked object properties and consumer interface records to a responseinterpreter system, and the process ends.

Referring now to FIG. 14, a high level logic flowchart depicts a processand program for response prediction from tracked 3D object propertiesand consumer interface records. In the example, the process starts atblock 1400 and thereafter proceeds to block 1402. Block 1402 depicts adetermination whether the response interpreter system receives 3D objectproperties. When the response interpreter system receives 3D objectproperties, then the process passes to block 1404. In addition toreceiving 3D object properties, the response interpreter system mayreceive consumer interface records.

Block 1404 depicts accessing a range of applicable definitions, and theprocess passes to block 1406. Applicable definitions may includebehavior definitions, product definitions, and voice stress definitions.Block 1406 illustrates the response interpreter system comparing the 3Dobject properties for tracked objects with the applicable definitions.Next, block 1408 depicts the response interpreter system detecting oneor more behavior definitions and product definitions with a closestmatch to one or more sets of 3D object properties. Thereafter, block1410 illustrates calculating a percentage probability that the 3D objectproperties match the selected definitions. Next, block 1412 depictspredicting from any predicted products and any consumer interfacerecords at least one stimulus within the consumer environment to whichthe consumer behavior is responsive. Thereafter, block 1414 illustratesgenerating at least one predicted behavior record with any predictedbehavior, percentage probability that the predicted behavior is correct,percentage probability that the predicted behavior is potentiallyadverse, and other sensed data. Next, block 1416 depicts transmittingthe predicted response records to a particular response-enabledapplication, and the process ends.

With reference now to FIG. 15, a high level logic flowchart depicts aprocess and program for applying a predicted behavior record in aresponse-enabled consumer interaction system. As illustrated, theprocess starts at block 1500 and thereafter proceeds to block 1502.Block 1502 depicts a determination whether a response-enabled consumerinteraction system receives a predicted response record. When theconsumer interaction system receives a predicted response record, thenthe process passes to block 1504. Block 1504 illustrates collectingother data associated with the consumer environment. Next, block 1508illustrates translating the predicted response records and othercollected data into one or more control signals for attempting to adjustthe consumer's experience within the consumer environment. Thereafter,block 1510 illustrates controlling output of the control signals to theselected output controllers, and the process ends.

While the invention has been particularly shown and described withreference to a preferred embodiment, it will be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the invention.

1. A computer-implemented method for adjusting a consumer experiencewithin a consumer environment, comprising: capturing a three-dimensionalmovement of the consumer within the consumer environment, wherein thethree-dimensional movement is determined using at least one imagecapture device aimed at the consumer; identifying at least one behaviorof the consumer in response to at least one stimulus within the consumerenvironment from a three-dimensional object properties stream of thecaptured movement; detecting whether the at least one behavior of theconsumer indicates a type of response to the at least one stimulusrequiring adjustment of the consumer environment by comparing the typeof response indicated by the at least one behavior with a plurality ofresponse rules and identifying at least one particular response ruleapplicable for the type of response to the at least one stimulus; andresponsive to detecting that the behavior of the consumer indicates atype of response to the at least one stimulus requiring adjustment ofthe consumer environment, generating a control signal according to theparticular response rule to trigger at least one change of the at leastone stimulus within the consumer environment.
 2. The method according toclaim 1, wherein capturing a three-dimensional movement of the consumerfurther comprises capturing the three-dimensional movement using astereoscopic image device to identify and track a particularthree-dimensional movement of the consumer.
 3. The method according toclaim 1, wherein capturing a three-dimensional movement of the consumerfurther comprises capturing the three-dimensional movement using atleast one stereoscopic image device and at least one sensor enableddevice to detect content accessible to a consumer at an interface withinthe consumer environment, wherein the content is the stimulus within theconsumer environment.
 4. The method according to claim 1, whereinidentifying at least one behavior of the consumer in response to atleast one stimulus within the consumer environment from athree-dimensional object properties stream of the captured movementfurther comprises calculating a percentage probability that the capturedthree-dimensional movement represents a particular behavior defined in aplurality of behavior definitions.
 5. The method according to claim 1,wherein identifying at least one behavior of the consumer in response toat least one stimulus within the consumer environment from athree-dimensional object properties stream of the captured movementfurther comprises identifying at least one of body movement of theconsumer comprising a fine granularity movement of at least one of afacial expression, an eye movement, a muscle contraction, and change inskin surface characteristics of the consumer within thethree-dimensional object properties stream matching at least one of aplurality of behavior definitions.
 6. The method according to claim 1,further comprising: responsive to detecting the consumer within theconsumer environment comprising at least one of a first selected areacomprising a point of sale terminal, a second selected area comprising adisplay within a store, a third selected area comprising a kiosk withina store, and access to an online store interface, capturing thethree-dimensional movement of the consumer within the consumerenvironment.
 7. The method according to claim 1, wherein identifying atleast one behavior of the consumer in response to at least one stimuluswithin the consumer environment from a three-dimensional objectproperties stream of the captured movement further comprises:identifying at least one behavior responsive to the at least onestimulus of a particular product by detecting at least one of theconsumer glancing at the particular product, the consumer picking up theparticular product, the consumer reading a label affixed to the product,the consumer reading the price tag for the product, and the consumerhesitating while interacting with the product.
 8. The method accordingto claim 1, wherein generating a control signal according to theparticular response rule to trigger at least one change of the at leastone stimulus within the consumer environment further comprises:generating the control signal according to the particular response ruleto trigger a program controller to adjust the content displayed on aninterface within the consumer environment.
 9. The method according toclaim 1, wherein generating a control signal according to the particularresponse rule to trigger at least one change of the at least onestimulus within the consumer environment further comprises: generatingthe control signal according to the particular response rule to triggera store attendant system to prompt at least one store attendant withinthe consumer environment to provide additional aid to the consumer. 10.The method according to claim 1, wherein generating a control signalaccording to the particular response rule to trigger at least one changeof the at least one stimulus within the consumer environment furthercomprises: generating the control signal according to the particularresponse rule to trigger running an additional computer systemassociated with the consumer environment to provide a service to theconsumer.
 11. A system for adjusting a consumer experience within aconsumer environment, comprising: a response processing systemcommunicatively connected to a network, further comprising: means forcapturing a three-dimensional movement of the consumer within theconsumer environment, wherein the three-dimensional movement isdetermined using at least one image capture device aimed at theconsumer; means for identifying at least one behavior of the consumer inresponse to at least one stimulus within the consumer environment from athree-dimensional object properties stream of the captured movement; acustomer interaction system communicatively connected to the responseprocessing system via the network, further comprising: means fordetecting whether the at least one behavior of the consumer indicates atype of response to the at least one stimulus requiring adjustment ofthe consumer environment by comparing the type of response indicated bythe at least one behavior with a plurality of response rules andidentifying at least one particular response rule applicable for thetype of response to the at least one stimulus; and means, responsive todetecting that the behavior of the consumer indicates a type of responseto the at least one stimulus requiring adjustment of the consumerenvironment, for generating a control signal according to the particularresponse rule to trigger at least one change of the at least onestimulus within the consumer environment.
 12. The system according toclaim 11, wherein the means for capturing a three-dimensional movementof the consumer further comprises means for capturing thethree-dimensional movement using a stereoscopic image device to identifyand track a particular three-dimensional movement of the consumer. 13.The system according to claim 11, wherein the means for capturing athree-dimensional movement of the consumer further comprises means forcapturing the three-dimensional movement using at least one stereoscopicimage device and at least one sensor enabled device to detect contentaccessible to a consumer at an interface within the consumerenvironment, wherein the content is the stimulus within the consumerenvironment.
 14. The system according to claim 11, wherein the means foridentifying at least one behavior of the consumer in response to atleast one stimulus within the consumer environment from athree-dimensional object properties stream of the captured movementfurther comprises means for identifying at least one of body movement ofthe consumer comprising a fine granularity movement of at least one of afacial expression, an eye movement, a muscle contraction, and change inskin surface characteristics of the consumer within thethree-dimensional object properties stream matching at least one of aplurality of behavior definitions.
 15. The system according to claim 11,the response processing system further comprising: means, responsive todetecting the consumer within the consumer environment comprising atleast one of a first selected area comprising a point of sale terminal,a second selected area comprising a display within a store, a thirdselected area comprising a kiosk within a store, and access to an onlinestore interface, for capturing the three-dimensional movement of theconsumer within the consumer environment.
 16. The system according toclaim 11, wherein the means for identifying at least one behavior of theconsumer in response to at least one stimulus within the consumerenvironment from a three-dimensional object properties stream of thecaptured movement further comprises: means for identifying at least onebehavior responsive to the at least one stimulus of a particular productby detecting at least one of the consumer glancing at the particularproduct, the consumer picking up the particular product, the consumerreading a label affixed to the product, the consumer reading the pricetag for the product, and the consumer hesitating while interacting withthe product.
 17. The system according to claim 11, wherein the means forgenerating a control signal according to the particular response rule totrigger at least one change of the at least one stimulus within theconsumer environment further comprises at least one of means forgenerating the control signal according to the particular response ruleto trigger a program controller to adjust the content displayed on aninterface within the consumer environment, means for generating thecontrol signal according to the particular response rule to trigger astore attendant system to prompt at least one store attendant within theconsumer environment to provide additional aid to the consumer, andmeans for generating the control signal according to the particularresponse rule to trigger running an additional computer systemassociated with the consumer environment to provide a service to theconsumer.
 18. A program product comprising a storage-typecomputer-usable medium including a computer-readable program foradjusting a consumer experience within a consumer environment, whereinthe computer-readable program when executed on a computer causes thecomputer to: capture a three-dimensional movement of the consumer withinthe consumer environment, wherein the three-dimensional movement isdetermined using at least one image capture device aimed at theconsumer; identify at least one behavior of the consumer in response toat least one stimulus within the consumer environment from athree-dimensional object properties stream of the captured movement;detect whether the at least one behavior of the consumer indicates atype of response to the at least one stimulus requiring adjustment ofthe consumer environment by comparing the type of response indicated bythe at least one behavior with a plurality of response rules andidentifying at least one particular response rule applicable for thetype of response to the at least one stimulus; and generate a controlsignal according to the particular response rule to trigger at least onechange of the at least one stimulus within the consumer environment. 19.The program product of claim 18, wherein said computer-readable programis transmitted over a network.
 20. The program product of claim 18,wherein the computer-readable program when executed on a computerfurther causes the computer to capture the three-dimensional movementusing a stereoscopic image device to identify and track a particularthree-dimensional movement of the consumer.